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Abstract 

This report details an ongoing investigation of the decision-making processes of a group of secondary school 
students in south-eastern Australia undertaking information search tasks. The study is situated in the field of 
information seeking and use, and, more broadly, in decision making. Research questions focus on students’ 
decisions about the relevance and reliability of information. Data collected include video screen captures, 
think-aloud reports, and interviews. Qualitative data analysis developed a preliminary grounded theory to 
describe decision-making processes. Students depended on system-provided relevance cues, rejected 
particular resource categories, examined remaining items for general and specific relevance, and primarily 
used a process of corroboration to assess reliability. Selected implications for educators are raised. 

Introduction 

This paper reports progress in a study of selected aspects of the information search process (ISP) of a group 
of secondary students in an Australian school. The ISP is defined as ‘the user’ s constructive activity of 
finding meaning from information in order to extend his or her state of knowledge on a particular problem or 
topic’ (Kuhlthau, 1991, p. 361). The study focuses on the criteria applied by students when deciding on the 
relevance and reliability of found information. Student information behaviour is investigated within a 
framework of problem solving, currently a dominant approach in information seeking research (Spink & 
Cole, 2006). Martinez (2006) identifies problem solving as a major category of metacognition. 

The learning context of my study is a task in which students draw information from a variety of sources to 
develop and demonstrate their understanding of a topic. The study views the user as an ‘active, constructive 
person’; acknowledges an individual, subjective view of information where ‘information is the sense or 
internal knowing a person constructs to fill an information gap’; and recognises the ‘unique psychological, 
sociological, cognitive factors that shape information seeking and use’ (Todd, 2003, p. 29). Students’ 
dynamic and personal interaction with information indicates qualitative methods of investigation, with the 
study situated in the constructivist paradigm. 

The overarching objective of this study is to identify selected strategies used by secondary school students, 
as demonstrated in the decisions they make about their learning. Two specific research questions arise from 
this objective: 

• What decisions do students make about the relevance of information? 

• What decisions do students make about the reliability of information? 

My investigation includes a view of information seeking and use from the perspective of decision making. 
Warwick, Rimmer, Blanford, Gow, & Buchanan’s (2009) longitudinal study of undergraduate students 
explored information seeking in the context of decision making theory, particular satisficing behaviours; 
however, its primary focus was the development of expertise in information behaviour over time. Tanni & 
Sormunen (2008) point to the scarcity of research into the information seeking and retrieval behaviour of 
school students. Branch (2003) claimed that adolescent information seekers are not becoming more effective 
and recommended more research into the experiences of adolescents in electronic information-seeking 
environments. Chi & Ohlsson (2005) claim that we have a poor understanding, in general, of the ways in 
which students acquire declarative knowledge. My study is intended to contribute to discussions about 
credibility in studies of information seeking and retrieval, which. Rich & Danielson (2007) report, are of 
recent interest. Watson (2004) supports a qualitative approach when contending that, with the focus of 
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schools on gathering quantitative data about student performance, educators ‘rarely listen to students’ talk 
and reflections on what they know or how they learned it’ (p. 145). 



Related literature 

Defining a decision 

A typical definition of a decision, from classical decision theory, is that it is ‘a choice among alternatives that 
will yield uncertain futures, for which we have preferences’ (Howard, 2007, p. 40). Newell, Lagnado & 
Shanks (2007) define a decision as ‘a commitment to a course of action’, distinguishing it from a judgement, 
‘an assessment or belief about a given situation based on the available information’ (p. 19). Betsch and 
Haberstroh (2005) conclude that ‘decisions are no longer viewed as singular or isolated events in time, but as 
being embedded into the stream of the individual’s experience’ (p. 361). 

Defining relevance 

Kelly (2005) contends that ‘The concept of relevance ... is necessarily a part of any theory of information- 
seeking and retrieval’ (p. 169). Despite variations in definitions of relevance (Cosijn & Ingwersen, 2000; 
Jacso, 2006; Otterbacher & Radev, 2008; Saracevic, 2007; Spink, Greisdorf & Bateman, 1998), Borlund 
(2003) asserts that recent renewed interest has resulted in a ‘consistent and compatible understanding’ of the 
concept of relevance, culminating in a phenomenon that can be ‘approached conceptually and operationally 
from the user’s perspective’ (pp. 913-914). 

Kelton, Fleischmann and Wallace (2008) assert that relevance ‘measures the degree to which the information 
matches the requirements of the user’ (p. 370); this implies a relational nature, and a situational aspect, given 
implicit changes in the user’s requirements. Vakkari and Hakala’s (2000) more detailed definition shows this 
explicitly: 

Relevance is understood as the relation between the actor’ s information need and the information in the documents 
that contribute to the solution of his or her information problem .... [T]he relevance of documents is assessed by the 
actors in terms of their support and contribution to a certain task. This view of relevance can be called situational or 
task-embedded, (p. 541) 

HJprland (2000) claims that it is easier to define non-relevance than relevance and lists eleven examples of 
situations in which a user would judge a document to be not relevant. An approach emphasising non- 
relevance may more accurately reflect Baran and Davis’s (2006) point that, in our screening out of irrelevant 
information, we mostly avoid information. 

Relevance in studies of school students 

Most studies of students’ relevance assessments are of undergraduate and graduate students. Hirsh’s (1999) 
study is one of few with an explicit focus on the criteria that school-age students use for assessing relevance. 
Hirsh identified nine categories of relevance assessments applied to textual material by ten Grade 5 students 
from Tucson, Arizona: authority, convenience/accessibility, interesting, language, novelty, peer interest, 
quality, recency/temporal issues, and topicality. 

Tanni and Sormunen’s (2008) literature review of students’ information behaviour in relation to assigned 
tasks, in light of the paucity of research of school students, broadened their scope. Their review addresses the 
association between information seeking style, ISP stage and relevance criteria. Tanni and Sormunen 
borrowed Heinstrom’s (2002) typology of information seekers: fast surfers, broad scanners, and deep divers. 
Fast surfers based their relevance judgements on ‘superficial criteria’, depending on ‘outer features rather 
than information content’; they also selected documents that confirmed their pre-existing knowledge 
(Heinstrom, 2002, p. 216). Broad scanners searched widely, accessed many sources, and exhibited sound 
skills in critical analysis of information. Deep divers displayed depth in strategies employed in searching and 
aimed for depth in content of information sources; they gave authority considerable weighting in their 
relevance judgements, preferring sources written by those with ‘status and position within the field’ (p. 238). 
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Although relevance did not feature in her central research questions, Gross (2004) observed that 
only the older elementary school students noted issues related to relevance. Chung and Neuman’s 
(2007) research question to do with students gathering, selecting and organising information for their 
research assignments has implications for relevance. Twenty-one students from a Maryland school 
considered information relevant when aligned to the topic for their persuasive speeches and regularly chose 
material that supported their pre-existing views. Chung and Neuman observed what they termed ‘functional 
relevance’ when students focused on finding the four types of information mandated in the assignment 
requirements. Students relied on systems relevance when depending on relevance ranking supplied hy Weh- 
hased sources: 

Students believed that the electronic sources performed the intellectual tasks that they had to do for themselves with 
print sources: filtering through and sorting out the relevant information was perceived as the biggest advantage. 

(Chung & Neuman, 2007, p. 1511) 

Defining reliability 

The current study does not draw distinctions between credibility, reliability, accuracy, authority, and quality 
of information; and trust in information. Rich and Donaldson’s (2007) literature review notes that 
information science generally conceptualises credibility as a subset of relevance judgements. Wathen and 
Burkell (2002) cite sources that use credibility to mean believability, and that describe credible sources as 
trustworthy and having expertise. Kelton et al. (2008) consider reliability and confidence as narrower 
concepts than trust, emphasising the subjectivity of the information seeker. Robins, Holmes & Stansbury’s 
(2010) definition of credibility also suggests its subjectivity: ‘the perception of trustworthiness, believability 
or expertise perceived in the stimuli shown to subjects’ (p. 14). 

Studies of reliability 

Metzger’ s (2007) literature review reports that the overriding consideration in credibility evaluations is 
design of Web resources. Wathen and Burkell (2002) summarise variables in credibility assessments of 
computer-based media; they refer to the role of surface aspects, listing examples: ‘appearance of the site’s 
homepage (colors, graphics, etc.), the usability of the interface, how well the information is organized’ (p. 
140). Suggesting intuitive decision making, Robins and Holmes (2008) refer to subjective, or viscerally 
based, assessments of properties such as ‘the combination of colors, layout, overall aesthetic treatment, fonts, 
use of bulleted lists, or presentation of tabular data’ (p. 390). Their study of tertiary students found that 
perceptions of higher aesthetic quality were associated with higher credibility rankings. 

Prabha, Connaway, Olszewski and Jenkins (2007) found that university faculty used perceived credibility, or 
trustworthiness, as a cue to ending their information search. The researchers considered role and rational 
choice theories, with the latter’s offshoot of satisficing, in their work. 

The few studies of school-age children referring to reliability include Hirsh’s (1999), in which a mere 2% 
used the authority of information for deciding relevance; and Shenton and Dixon’s (2003), in which none 
reported instances of evaluating credibility of non-print information. 

Methodology 

Participants 

Thirty-four students in Years 9 to 11 (14- to 17-year-olds) in an independent school in south-eastern 
Australia volunteered to take part. There was no overlap in participants between the first and second phases 
of data collection; six of those from the second phase, however, also participated in the third phase. The 
students were mostly high academic achievers. They were English-speaking and there was considerable 
homogeneity in the students’ cultural backgrounds. The students were engaged in research tasks set by 
teachers in a range of curriculum areas, including Humanities (particularly History), Science, and Music. 
Students were given considerable choice of topics to pursue. The group had common experience of a process, 
a typical starting point for a grounded theorist (Cresswell, 2007). 
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Data collection 

In the first phase (16 students), data were sourced from semi-structured, one-to-one interviews (see 
Appendix A for topics); students’ journals of their ISP; and two questionnaires (adapted from Kuhlthau & 
Todd, 2007), both asking what students were finding easy to do and what was challenging at the respective 
points of topic selection and midway through the ISP. 

In the second phase (17 students, all of whom were different from those who had taken part in the first 
phase), one-to-one interviews were again used. The interview entailed working through a timeline of 
students’ research tasks; questions (adapted from Whitmire, 2003) included those in Appendix A. 

In the third phase of data collection (seven students, six of whom had taken part in the second phase), 
concurrent/think-aloud reports and retrospective reports/interviews were complemented hy video screen 
captures of online activity. Participants were asked to think aloud while working online for 20 minutes on a 
school assignment of their choice. They were asked for a commentary of their thoughts and feelings with as 
little silence as possible. Students were alone for 20 minutes with the task, after which the retrospective 
interview took place (in the course of the shared viewing of the screen recording); my question topics appear 
in Appendix A. 

The think-aloud report captured the normal and spontaneous behaviour of the participants while any 
structure imposed in the subsequent interview, based on the video-recorded material, came from my research 
questions. I considered a narrative interview inappropriate for its focus on a central event in the interviewee’s 
life (the ‘single overall narrative’ of Flick, 2006, p. 187) expressed in ‘his or her own spontaneous language 
in the narration of events’ (Jovchelovitch & Bauer, 2000, p. 61) over an hour or more. 

Omodei, McLennan and Wearing (2005) note three advantages to the own-point-of-view perspective, which 
they achieved using a head-mounted camera and I by video capture of screen activity. First, the perceptual 
field’s faithful reproduction in the replay acts as ‘a maximally-powerful stimulus to the spontaneous 
recollection of those mental events which occurred while the recording was being made’ (p. 275). The 
participant is less intent on presenting a positive personal image. Second, the reduced self-awareness or self- 
consciousness, brought about by not seeing oneself in the video replay, lowers the chances of self-protective 
behaviour or recollections selected by the participant to align with his or her preconceptions of what the 
investigator may be anticipating. Third, self-consciousness is minimized in the initial recording, thereby 
increasing the probability that the participant’s behaviour is as close as possible to the natural, spontaneous 
behaviour that might be expected without the recording process. 

While possible to observe decision outcomes, traditional observation methods would not have uncovered the 
psychological processes leading to those outcomes. The process of selecting what was relevant and reliable 
could not be readily discerned by observation, except by asking the student, in which case the encounter 
becomes a mini-interview. Such an interview would lack detail and interrupt the student’s thought processes. 

Hogarth (2005) concluded that ‘deliberation changes expressed preferences if subjects are unaware of the 
origins of those preferences’ and ‘decision aids that force people to be explicit about reasons for their 
decisions heighten satisfaction in choices’ (p. 75). The students in my investigation readily supplied reasons 
for their choices and there was consistency in answers across participants. Concurrent verbalisations, and 
retrospective interviews immediately following the task, may be assumed to elicit data about decision- 
making processes that are not substantially affected by their verbalisation. 

Data analysis 

I undertook an inductive analysis of gathered data in which categories emerged from the data. Dey (1999) 
calls this ‘heuristic coding’, which is ‘a characteristic of interpretive inquiry where the meanings of actors 
(and action) cannot be established in advance of the research’ (pp. 123-124). When considering the creation 
of a new category, I compared and contrasted the data that would be in the proposed category with the data 
in other categories, in order to ensure distinctions were maintained. This is part of the ‘constant comparison’ 
with which grounded theory has become synonymous. 
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In the first step of analysis, I assigned thematic categories, or ‘free nodes’ in QSR International’s (2008) 
NVivo qualitative data analysis software, to any phenomena in the data I considered related to my research 
questions, hut also noting other interesting segments of data, in case they provided evidence for or against an 
emerging theory. The second stage of analysis involved the allocation of ‘tree nodes’ in NVivo, clustering 
themes revealed in the free nodes. In a third stage of analysis, I found that some tree nodes were able to he 
analysed at a finer level and so developed categories to cover these. 

From data collected from the seven students in the third phase, I sketched, in rough flowcharts, the decision- 
making process of each. I used these to develop a generalised view of their processes, supplementing that 
view with data collected in earlier phases. 

Findings 

Before an individual information source was accessed, participants noted the category to which an item 
belonged; membership of particular classes led to rejection of the item. Sources that ‘passed’ this pre- 
relevance judgement stage were judged for relevance. Participants decided relevance on the basis of system- 
provided data, basing decisions primarily on the correspondence between keywords in the search string and 
keywords in the information source. Their judgements of reliability, apparently secondary to those of 
relevance, were based on information that could be corroborated, and on assessments of the author, writing 
style, and graphic design of the information. (See Figure 1. The linear and logical progression suggested by 
Figure 1 does not capture the recursive nature of the process or that students may apply the process to one or 
a set of information sources in no particular order.) 



© 2010 lASL, SLAQ and therein by the authors. Diversity Challenge Resilience: School Libraries in Action Proceedings of the 12* 
Biennial School Library Association of Queensland, the 39* International Association of School Librarianship Annual Conference 
incorporating the 14* International Forum on Research in School Librarianship, Brisbane QLD Australia, 27 September - 1 October 
2010 . 




Judgement 



Focus of Judgement 



Decision 



Outcome of Decision 



Pre-re levanoe 
luogement ol type 




Reputatioo of 
Intormaiwn sources 




Contains relevant 
or reliabe 
mlofmatlon? 




of infomnalion 
source 




a& group 
(preconceptions) 









Discard Irrelevant 
or unre iable 
sources 



Judgement of 
system -provided 
aa\a about 
information sources 




Metadata 



Order of results 



Which results have 
highest density of 
search terms with 
highest collocation? 



Which results 
occupy pnme 
positions (first 
result / others on 
first page)? 



Judgement of 
individjal 
information source 



Keywords within 
specific information 
source 



Headngs 



F-rst 

parap'apns 



Tope 
servo 'tees 



What is the level of 
connectedness of 
information source 
to topic? 



Reliability criteria 




Retain sources 
remaining afier 
filtering 



Retain results with 
highest density and 
those occurring 
early In me results 
let 



Discard 



Retain information 
sources wim best nt 
to topic 



Correspondence 




with prior 








knowledge, 




Low 




iiTtpressions of 








author, writing style. 








design 




High 





Discard 



Take notes; use in 
llnal information 
product 



Figure 1. Provisional model of students’ decision-making process 



Pre-relevance judgements 

Simultaneous with the examination of the metadata of search results, participants made pre-relevance 
judgements of information sources based on their preconceptions of the usefulness of certain categories of 
information source. This eliminated any information sources not meeting a participant’s baseline 
requirements, e.g., eBay and Amazon results, and an abstract of a journal article (Mary-Ann); websites in 
unfamiliar foreign languages (Paul); WikiAnswers (Chloe, who noted a progressive deterioration in quality of 
answers to her query); and video material (Sharon; Chloe). Source reputation played a part with respect to 
groups of information sources. 



Relevance from information system 

An information system such as Google provides metadata about the resources it retrieves. Information about 
the source was important to participants before examining specific information sources themselves. System- 
provided metadata included titles and snippets, or extracts, from each source. Karen described the snippets as 
‘catchers’ and ‘eye-grabbers’. A scientist’s name in the title of a source drew Mary-Ann’ s attention. Paul 
looked for a site whose URL used the name of the subject of his search, Beethoven. Participants relied on the 
density of search terms highlighted by bolding, and how closely they were collocated, to gauge the degree of 
relevance to their information needs. 
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Participants considered, before examining individual sources, order of results, taking it as a cue to relevance. 
A source listed first was regarded as most relevant by virtue of its prime position and was likely to be 
considered first. Participants assumed that the first result would contain the highest density of keywords. 
Karen sought relevance in the results of the first page, finding it too time-consuming to go beyond that. She 
matched words in the titles of search results with words in her search string, both for Internet and online 
library catalogue searches. Mary-Ann found it better to use fewer words in a search string because too many 
words gave results of lower relevance (‘vague things’)- She also used results from the first page of search 
results, noting increasing vagueness beyond that. 

Relevance in sources accessed 

After considering system-provided suggestions of relevance, participants considered the actual sources. The 
arrangement of information in a source was important to participants, who used headings, topic sentences 
and initial paragraphs in relevance judgements. 

Mary-Ann used headings to gauge correspondence between words in the information source and words in 
her topic. In particular, she sought a combination of a physicist’s name and references to greenhouses. The 
text’s first paragraph was also used when determining relevance. Victoria scanned the first sentence of each 
paragraph and continued reading when the initial sentence was relevant. Chloe’s experience suggested that 
easily accessed information facilitated relevance judgements: ‘with the books I was finding what I wanted 
when I wanted it’ . Chloe had high topic knowledge, having completed her book-based research before 
consulting online resources. This may explain why she felt that, in the less organised world of information on 
the Internet, relevance was less defined. 

When accessing individual information sources, participants sought an item of general relevance (i.e., 
offering a general treatment or overview of their topic) before looking for one of specific, or focal, relevance 
(i.e., fitting a chosen focus within the topic area), particularly if they lacked prior knowledge. A general 
treatment was desired for its introduction to the topic and, once assimilated, the participant proceeded to an 
item with focal relevance. The later matching of focal content to general content (mainly single items, and 
predominantly sourced from Wikipedia) would become important in corroborating information. Victoria 
described generally relevant information as offering a summary point. She then matched each summary point 
with information under current consideration to conclude that it was the ‘same stuff I’ve read in books and 
Wikipedia'. A search for overview information was particularly important to Victoria, who considered her 
prior knowledge of McCarthyism to be low but was later able to focus on the Wheeling speech. 

Relevance decisions had a ‘snowballing’ effect: they influenced what was noted in subsequent information. 
Mary-Ann spoke of the ‘on-flow effect’ , in which one piece of relevant information led to another. In 
building knowledge, Mary-Ann described ‘trying to find out something from something else’, a strategy in 
which ‘you take that bit of information and . . . see. . . what that can lead to’. This action is allied to using an 
overview of a topic; both the overview and the subsequently found bits of information form part of a chain of 
linked information. The approach is illustrated in Mary-Ann’ s words: ‘I use a piece of information from a 
good site and search on that.’ Relevance begets relevance. 

Reliability criteria 

Having accessed an individual information source and established its relevance, participants occasionally 
applied tests of reliability. It is not easy to disentangle decisions about relevance from those about reliability; 
for example, when a class of information sources is rejected, it is unclear whether this resulted from 
perceptions of irrelevance or unreliability, or both. I have treated relevance and reliability separately; 
however, they overlap and sometimes are broader judgements of information quality. 

The main test of reliability entailed corroboration: of one source with another (typically a general overview 
of the topic with one that was more focused); and of one source with the participant’s prior knowledge, 
which occasionally included personal opinion. An extract from Kelvin’s journal illustrates the symbiosis 
between Google and Wikipedia (further data extracts are in Table 1): 
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As with most research, I find the easiest and most effective way to do this [is] to be reading the Wikipedia articles on 
the subjects. After I have done this, I will Google search the subjects in order to test the accuracy of the information 
on Wikipedia. 

Table 1. Data extracts illustrating use of corroboration 

Corroboration of newly found information 

Brian: I look for the page author and if it’s something like a date or something that’s really common. I’ll be able to go 
and search it again and maybe find it on another page and that’ll affirm that that’s the correct date. 

Daniel: I had a look at other sites and then made sure that it was actually relevant and true, yeah, ’cause I’ve been told in 
a lot of other subjects not to use it [Wikipedia}, then I make sure that it’s correct ... I just sort of get a lot of opinions 
from the sites, so if I get one idea about something, then I have a look at another website that says the opposite of that, 
then I’ll try and find a few other sources to back up one or the other to find out which one is more relevant and true. . . . 

In most searches that I do I find a Wikipedia page as one of the first choices. They are not the most reliable pages 
although they do provide a lot of data so I use them as a reference and then back up the information with further 
research. 

Kelvin: I’d just kind of put ... in ... the Google search of it to kind of, I dunno, it depends what it, if it’s just kind of a 
value I think will just kind of, if it says, you know, the speed of sound or something I’ll look at . . . and then I’ll just kind 
of put that into Google and see what it says and just see if the two, see if kind of across a couple of sites, if all the values 
match. 



Corroboration of pre-existing knowledge 

Lily: I guess just compare them with what we think to see if they’re actually kind of reliable ... it’s only the opinions of 
people who made like valid suggestions and comments that we really kind of took into account. 

Noah: A lot of them you can kind of just tell because you can generally, with a bit of background information, you just 
know whether they sound feasible or not. 



Corroboration of newly found information and pre-existing knowledge 

Vicki: I’ll double-check it with, maybe a book, or another Internet website, or just on my knowledge so ... I’ll always 
double-check it to make sure it’s right. 



Other tests of reliability included the reputation of the source (which overlaps with the relevance of groups of 
resources, discussed above), URL details, author’s credentials, design, and style of writing. Participants 
trusted books as a class of resource, with their publishing process cited regularly as the source of their 
reliability. Paul considered books ‘really reliable’ because ‘they have to get past the publisher, so someone 
has to actually have read through the content, and given a tick of approval’ and had high regard for 
university-produced publications. Wikipedia was almost universally perceived as an unreliable though useful 
source; students, reporting that many teachers questioned its dependability, nevertheless used it for the 
framework it provided for a topic on which their background knowledge was low or lacking. Emily’s 
comments illustrate most participants’ connection with Wikipedia, whose widespread use, particularly as a 
launching point, is associated with guilt at not following teachers’ advice: 

Wikipedia, I use a lot, but that’s more to find, I don’t really use it for specific things because they say that you can 
change it and it’ s not really that reliable, so I use it to get an overview of the topic and then I go into Google and 
search for whichever bit I’m interested in ... I know I’m not supposed to, but I always use it [Wikipedia] to get a 
summary of the things . . . They [the teachers] said we’re not allowed to put it in our bibliography because it’s not a 
reliable site. I think it’s really useful ’cause it has pretty much everything in one spot but, yes, I don’t use it for getting 
specific information for my presentation, just an idea of it. 

In another example of source reputation used as a criterion, Charlotte, Sophie and Emily rated Encyclopaedia 
Britannica highly. Charlotte claimed that ‘there’s going to be a difference between, you know, using 
something like Britannica and, you know, Wikipedia is always a bit shifty’ . Sophie had Encyclopaedia 
Britannica at home and considered it a quality resource worth using. Emily aimed to 
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find sites that were more trust-, trustworthy maybe like, um, there’s like the Britannica Encyclopaedia [sic], things 
like that, things that were not somebody’s personal web page but were sort of a company ... a bit more trusted. 



Participants believed that URLs containing .gov, .edu and .org were reliable. Related was their perception 
that sophisticated sounding website titles gave the source some credibility. Mary- Ann found a ‘smarter- 
sounding website name’ alluring; she rejected a personal, even idiosyncratic, website; and bypassed a 
Wikipedia site and a Fox News site in her quest for a more scientific approach to the physics of greenhouses. 
Kristian said, of a site he used, that ‘it was a .gov site, so that usually does mean that it is reliable’. Richard 
similarly trusted government websites: ‘a few of them were from the War Memorial and I thought that those 
were the best sort of ones because they’re from the government and they’re a bit more established than 
Wikipedia’ . 

Author’ s credentials appeared to be important to participants but the data provided few examples of students 
actually investigating the qualifications of the author. One student concluded from the style of writing alone 
that the information had been written by a professor or, at least, by someone with expertise in the area. The 
lack of author details in a Wikipedia article gave Paul cause to regard another website, with its author’ s name, 
as more reliable. Displaying scepticism about its authorship, Rachel avoided relying on Wikipedia for 
statistical information: ‘When I actually get information for the speech, like, if there was any statistics or 
anything, I try not to use them from Wikipedia, ’cause it tT just like any old bum can write it.’ The relative 
informality of blogs and chat rooms caused distrust of their information, being described as ‘not official’ and 
being written ‘from the top of their head, not any sort of effort’ (Chloe). Richard commented that books are 
generally written by ‘people who know what they’re talking about’. 

Participants referred to the negative impact that poor design had on credibility. Unconventional choices of 
typeface and jarring colours in website design were enough to discourage a participant from fully trusting the 
information. For Richard, poor site design indicated a lack of effort: ‘they don’t look like there’s much 
design put into them ... so I look at them and I think, well, that’s probably not the best one to choose’ . A 
website lacked sophistication, in Charlotte’s opinion, if ‘you get little things popping up saying, oh, you 
know, ‘Press the joke button’, blah blah blah’. 

Participants were more inclined to believe information written in a style associated with academic writing. 
They were attracted to good argumentation; longer pieces of writing, which they associated with depth of 
topic treatment; and writing that provided citations and references. When authors presented a good argument, 
through the evidence marshalled, the source was considered trustworthy (Karen). Another attractive attribute 
was length (Karen: ‘eight separate pages just on the Roman slaves’). Mary-Ann saw appropriate depth in an 
encyclopedia website, a ‘major site’ with in-text references and a list of sources. When a source offered 
citations and further readings, Paul also perceived this as evidence of reliability. Jake used depth of topic 
treatment to identify useful sources: ‘how in depth it went into the treatment of the different races and just 
like how broad it was, like how good it was at painting a picture of the whole, sort of, scenario, what was 
going on’. The more extensive topic treatment attracted Elizabeth to books on her topic: ‘the Olympics, the 
websites, there wasn’t really that much information, but the books, I dunno, they just went through it more 
thoroughly’. Richard chose a website ‘that looks more like it’s had some detail and effort put into it.’ 
Dimensions of good writing, like other sets of dimensions, are not mutually exclusive. 

Reliability’s non-binary nature was illustrated by Paul’s hierarchy: he trusted the Australian Music 
Examinations Board to offer the most reliable information about Beethoven; a website such as 
lvbeethoven.com would, he thought, present information that was of intermediate reliability; and he regarded 
Wikipedia as the least reliable source. 

Discussion 

Pre -relevance judgements and relevance inferred from the information system may be termed ‘pragmatic 
relevance’, following the definition of Buckland, Butler, Kim, Norgard and Plaunt (1995): ‘the presence (or 
absence) of attributes specified by the searcher’ (p. 3). Order of results is a system-supplied cue in which 
primacy plays a substantial role. When determining relevance in sources accessed, students search for the 
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context in which keywords are used; intellectual engagement becomes evident, in which only the human, 
with his or her particular information needs, can discriminate between sources and parts of sources. 
Considered below are the roles of linking pieces of information, and the confirmatory bias, in corroboration. 
The process of corroboration, although conceptualised as a way of establishing the trustworthiness of 
information, could equally be considered ‘relevance chaining’ with the first source influencing perceptions 
of relevance in the second source. This underlines the occasionally weak distinction between relevance and 
reliability, uniting them as quality indicators. 

Primacy 

Order of search results is system-provided. Cognitive psychologists recognise the impact that first-presented 
information has on subsequent information processing. Gemsbacher (1990) found that ‘Comprehenders 
spend more cognitive capacity processing initial words and sentences than later-occurring words and later- 
occurring sentences’ (p. 9), calling this phenomenon ‘primacy’. Students in my study expended more time 
processing early-occurring search results than later-occurring ones; similarly, they spent more time on initial 
websites (those on the first page) than on ones that occurred later. Aula and Nordhausen’s (2006) survey 
concluded that Web users ‘typically only check one result page’ (p. 1678). As Google is the search engine of 
universal choice of this study’s participants, that search engine’s initial results commandeered students’ 
processing capacities. With a Wikipedia entry common on the first page of a set of Google search results, 
students’ cognitive attention was directed to that online encyclopedia. Like the students in Sundin and 
Francke’s (2009) study, those in the current study used Wikipedia to develop an overview of their chosen 
topic but were aware that it is not a respected source in the sociocultural context of the school. 

Early and easy solutions are frequently attractive to decision makers. This explains the appeal of Google and 
Wikipedia, and users’ habitual return to them. The primacy of first-presented or first-found information is 
likely to be reinforced by several other phenomena noted in decision making theory, e.g., that ‘information 
that is available is also perceived to be more frequent, probable and causally important’ (Newell et ah, 2007, 
p. 35). Fixation on an early solution may indicate a maladaptive problem solving strategy; however, it may 
lead to an outcome acceptable to the decision maker. A choice selected on a preliminary basis tends to 
assume primacy. In the case of participant Sophie, the selection of a book that met her immediate 
requirements effectively precluded the consideration, or at least reduced the chances of consideration, of 
other information. Although Svenson (2003) notes the primacy of a preliminary choice alternative, the author 
allows for the possibility that new preliminary choices may supplant old ones. Teachers should, through task 
design, encourage students to question more strongly the adoption of information found early in the ISP and 
to reflect on how information found later in the ISP compares. 

Operating in conjunction with primacy and the dominance of the preliminary choice is the stopping rule 
referred to by Gigerenzer (2004), a heuristic that proposes that the ‘search is stopped immediately after the 
first discriminating cue is found’ (p. 75). The principle may be stated informally as ‘Stop when you 
recognise information relevant to your information search need’. The stopping rule is related to Zipf’s (1949) 
principle of least effort, which posits that ‘an individual’s entire behavior is subject to the minimizing of 
effort’ (p. 6); an individual expends the least amount of energy required to perform a function. If the overlap 
between four major search engines is a low 1.1% for the first results page on the same queries (Spink, Jansen, 
Blakely & Koshman, 2006), then students are not availing themselves of the large quantities and diversity of 
available information resources. 

Information evaluation strategies 

Students evaluating the quality of information sources are employing critical thinking skills, identified by 
Martinez (2006) as a major aspect of metacognition. Compilers of information evaluation checklists appear 
to assume that the user evaluates each resource individually, although Auburn University Library (2003) 
refers to the verification of the source of information by consulting other sources. Meola (2004) criticises 
guidelines that focus on internal properties of a web site for not promoting critical thinking, and offering 
inadequate evaluative questions. In the criticised evaluative approaches, an individual resource is evaluated; 
this constitutes a resource -by-resource evaluation. However, my data suggest that evaluation in practice is 
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done on an inter-resource basis, i.e. one resource is compared with another in order to judge the reliability 
and credibility of the first. Many students will find it unnatural to avoid this critical thinking approach. 

Data on the type of information sources that were rejected by participants showed which categories of 
sources did not meet criteria for inclusion in a set of reliable resources. Their behavior followed Pirolli and 
Card’s (1999) principle of lost opportunity, in which ‘a class of items should be ignored if the profitability . . . 
for those items is less than the expected rate of gain ... of continuing search for other types of items’ (p. 655) 
Related is the decision strategy referred to as ‘elimination by aspects’ (EBA), which ‘eliminates options that 
do not meet a minimum cutoff value or do not have a desired aspect for the most important attribute’ (Payne 
& Bettman, 2004, p. 118). EBA may overlook some relevant and reliable information, such as that websites 
offering extracts from books may offer useful information. 

Making links with prior knowledge 

The use of existing knowledge as the foundation for new knowledge has pedagogical implications. 
Sometimes prior knowledge is freshly acquired, as in a just-read overview of a topic. Not only does 
background knowledge provide the ‘hooks’ for attaching new knowledge, it is crucial when establishing 
reliability through corroboration. Drawing on Gernsbacher’ s (1990) concept of a structure -building 
framework, students’ background knowledge forms the foundation of a mental structure, which they 
subsequently seek to develop. 

The ability to draw from knowledge in memory is an important component of students’ learning. 
Brainstorming; concept mapping, and other graphic organisers, such as outline templates and Venn 
diagrams; and note -taking (Peper and Mayer, 1986) help students to make links with their prior knowledge. 

In Herring’s (2006) report on 52 Year 8 students in a Yorkshire school, brainstorming and concept mapping 
aided students to focus on their prior knowledge. Chung and Neuman (2007) claimed an outline template and 
concept maps reduced cognitive load for students when organising their information. Brooks and Shell 
(2006) also point to the positive influence of concept mapping on student achievement. 

Confirmatory bias 

Study participants looked for correspondence with their own opinions or pre-existing knowledge. Such 
behaviour is explained, in terms of decision making theory, as the effect of the confirmatory bias, ‘the 
tendency to treat evidence consistent with one’s preexisting beliefs or personal goals more favourably than 
inconsistent evidence’ (Haines & Moore, 2003, pp. 265-266). When students draw on their prior knowledge 
and prior experiences, they are demonstrating an awareness of their knowledge states, and thus are exercising 
metamemory and metacomprehension skills referred to by Martinez (2006) as major aspects of 
metacognition. As in making links with prior knowledge, a search for confirmation is implicated in 
corroboration. 

Resonance with one’s preconceptions is what Eischer, Greitemeyer, and Erey (2008) call standpoint- 
consistent information. Their four studies, in which self-regulation resources were manipulated, showed that 
‘Participants with depleted self-regulation resources exhibited stronger confirmatory information processing 
tendencies than did nondepleted participants’ (p. 392) and that standpoint-inconsistent information taxes an 
individual’s self-regulation resources. Standpoint-consistent information is accorded higher credibility and 
importance (Eischer et al., 2008). 

Kelton et al. (2008) refer to an identification process, which ‘reflects the degree to which the information 
contained within a source conforms to the user’s own identity, goals, and values’, claiming that trust in the 
information is ‘enhanced if the user resonates with the style, arguments, or objectives presented’ (pp. 369- 
370). 

Chung and Neuman (2007) found that students often chose information according with their own views. 
Similarly, Seamans (2002) reported that all nine undergraduate students in an English class set out with a 
‘predetermined viewpoint they were looking to support, as opposed to more broadly exploring a topic’ (p. 
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116). Students should be encouraged to evaluate their prior knowledge like any other information, and be 
prepared to constantly compare it to new information as their knowledge develops in the course of their ISP. 



Conclusion 

Participants searching for information online, faced with decisions about how to respond to search results, 
made three types of selection judgements: pre -relevance judgements; relevance judgements; and reliability 
judgements. Pre-relevance judgements entailed assessing which classes of information source were unlikely 
to offer relevant and reliable information, and so these were rejected (demonstrating an EBA decision 
strategy); and which were worth investigating further. Relevance judgements of those information sources 
considered worth further examination were of two types: those made about the source, before accessing it, 
based on system-provided data; and those made about the source, once the source itself had been accessed. 
Participants applied tests of reliability, the chief of which was corroboration. However, perceived relevance 
appeared to override considerations of reliability. 

Three key learnings: 

• The comparison of information in one source with that in another forms part of naturalistic decision 
making and should be encouraged in information evaluation. 

• Wikipedia holds an ambivalent but entrenched position in the academic lives of students. While 
assignments should be designed to require students to seek introductions to topics from a wider variety 
of sources, students are cautious about Wikipedia information, helped by views absorbed from teachers. 

• The development of metacognitive skills has the potential to mitigate the negative effects of decision 
making that may arise from accepting ‘good enough’ solutions, as seen in the early adoption of a readily 
available solution, a high dependence on search engine relevance rankings, and the operation of the 
confirmatory bias. 
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Appendix A: Interview topic guides 

Phase 1 

• Factors that motivated students to choose their topic. 

• Elements of their research work with which students were most satisfied, and those that they found most 
challenging. 

• What students learned from doing the assignment, whether in relation to informational content or 
research techniques. 

• How students learned. 

• The level and type of assistance students received, or would like to have received, from teachers and 
teacher librarians. 

Phase 2 (Key questions) 

• What, or how much, did you know about the topic before beginning your research? 

• How did you decide which sources were good for your research assignment? What factors did you use to 
judge whether a source was good or not? 

• If you came across sources that you disagreed with, or two sources that contradicted each other, how did 
you handle that? 

Phase 3 

• Decisions the student had made about the relevance and reliability of information. 

• Topic choice. 

• Prior knowledge about the topic. 

• Feelings associated with the task. 
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